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INTRODUCTION 


Aircraft accidents in the United States which involve air 
carrier aircraft on either revenue, training or ferry flights 
receive a thoroughgoing evaluation conducted by an accident 
investigation team manned by experts from the National Trans- 
portation Safety Board (NTSB) . The team is supplemented by 
technical experts from the Federal Aviation Administration ( FAA) , 
other government agencies, airframe and engine manufacturers, 
the airline, the airline pilot group, and other interested 
parties. (Other aircraft accidents may also be investigated by 
the NTSB, but are not the subject of interest in this study.) 

The NTSB team performs its investigation at the site of the 
accident initially, arriving at the scene as rapidly as possible. 
Analysis of the information gathered at the scene and through 
other sources is carried out over a period of time lasting from 
days to months; results are presented to the Board and an 
official report of the accident is publicly released, assigning 
a cause or causes and contributing factors to the accident. 

The major classification of accidents by NTSB may be grouped 
into those factors related to mechanical failure (e.g. airframe, 
power plant, component), environmental (e.g. weather, airport, 
terrain), and those factors related to human actions (e.g. pilot/ 
personnel-human error) . Human error has been a consistently 
high category (•.n.is.u. 31 ) , with NTSB data relating cause factors for 
air carrier accidents (all operations) for the period 1964-1969 
indicating, for example, that pilot factor was assigned as one 
of the causal factors in 58 percent of fatal accidents ( 1 ) . 

In fact, 55 percent of U.S. air carrier accidents continue to 
involve "pilot/personnel" as a cause or factor element ( 6 ) . 
General aviation data are even more impressive with the pilot 
listed as at least partially responsible in as high as 83 per- 
cent of reported accidents ( 2 ) . Applying the 58 percent 
figure to the average of 30 U.S. air carrier accidents investi- 
gated annually by the NTSB, it is estimated that 17 of these 
are pilot error-related.* 

Accident prevention is generally identified as the primary 
objective of flight safety programs and policies. The primary 
goal of accident investigation is to unravel in a time sequence 
all the events which led up to an accident in order that some 
action can be taken toward prevention of similar accidents. 
Equipment or material which fails to perform the specific 
function for which it is designed can generally be traced and the 


*The problem of human error in maritime accidents is 
receiving increasing attention. The Human Error Panel of the 
Maritime Transportation Research Board, National Academy of 
Sciences, is currently exploring means of identifying the nature 
of human error contingencies in ship accidents ( 12 ) . 



cause of the accident can be stated with some certainty; e.g. the 
brakes failed because of loss of hydraulic pressure. There can 
be and usually are a host of "contributing" factors, each of 
which has an impact upon the primary cause of the accident. 

Other events are involved, particularly when no critical 
material failure occurs and a pilot's act of omission or com- 
mission could have prevented the accident. In an accident inves- 
tigation when no significant material failure is identified and 
the occurrence of many contingencies are involved, the accident 
is frequently classified: "primary cause - pilot error". Those 
involved in analyses of accident data were the first to recognize 
the limitations of using a single primary cause factor. The 
analysts compiled data and published analyses of situations, con- 
ditions, and places where major accidents occurred. Using this 
approach, many pilot error studies were published; for example, 
analysis of undershoot and overshoot accidents, midair collisions , 
and disorientation accidents. While this type of study contributes 
somewhat to the understanding of the factors involved and occa- 
sionally provides some indication of the corrective action to be 
taken, no organized or systematic approach has been used in re- 
lating the many contingencies which are hidden under the term 
"pilot error". 

To know that no equipment failed but that the pilot committed 
an error which caused or contributed to the accident is of little 
consequence unless action is taken to prevent the future occurrence 
of similar human error accidents. To place the blame on the pilot 
does little to prevent others from committing the same error unless 
design changes are instituted, procedures modified, or training 
programs altered to eliminate that type of human error. Early in 
accident investigation it was recognized that if similar accidents 
were to be grouped, a standardized data format was essential. It 
was also important to assign a primary cause factor so that like 
accidents could be categorized into distinct classes, e.g. human 
error accidents. From the standardization of definitions, the 
artificial grouping of accidents by various criteria and the 
assignment of causal factors arose some of the difficulties in 
making precise statements about human error accidents. When 
human error data are normalized or grouped into distinct classes, 
a determinable amount of specific information is lost in that 
transformation. It is easy to relate all human error accidents 
identified as those in which "pilot failed to use proper judge- 
ment". However, it is not easy to show the relationship of major 
situational constraints, equipment and system influences, and 
decision-making (or cognitive processes) which contributed to a 
"situation" resulting in the pilot's failure to use proper judge- 
ment. 

Review of the human error literature reveals distinct 
categories of accident studies. First, there are those publica- 
tions showing accident frequencies. These are "impact" studies. 

The reports show how many aircraft have been destroyed or the 
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number of accidents which resulted in major damage. They show 
numbers of fatalities or major injuries and the frequencies are 
distributed by type of aircraft, hours flown, pilot experience, 
time of day, length of runway, etc. Other types of publications 
include case histories in which a particular accident is de- 
scribed in detail. Others include study of specific problems 
or failures in aircraft subsystems which occurred during flight, 
focusing close attention on failures of hydraulics, navigation 
systems, etc. Insufficient emphasis has been directed to the 
study of human error and only a fraction of the money spent in 
altering aircraft systems or correcting material failures is 
invested in attempting to analyze and correct failures resulting 
from human limitations. 

There is a school of thought which holds that nothing can 
be done to prevent the human error type of aircraft accident and 
that data accumulated during accident investigations are of 
limited value for establishing research programs in this area. 

We feel, however, that accident data are indeed a valuable source 
of human factors information and these data should be extracted 
and made available to research agencies to assist in research 
planning. The solution to the problem of how to utilize accident 
data as a basis for human factors research is not simple. However, 
the problem can be stated concisely: Accident analysis data are 

not presently precise enough, and neither are they in an appro- 
priate form to be used for the prediction of future human error 
accidents, nor even for establishing criteria for human error 
research. There have been notable exceptions such as the studies 
which led to the relocation of communications equipment to pre- 
vent disorientation. The U.S. Air Force has attempted to approach 
the problem through the use of annual USAF Industry System Safety 
Conferences dealing with research needs. The U.S. Navy has 
attempted to study accident causes in the human factors field 
through the medium of the Human Error Research and Analysis 
Program (HERAP) ( 5 , 16) . A longitudinal systems approach related 
mission and system requirements to human error. The purpose of 
the program was "to obtain and use appropriate information con- 
cerning human engineering as related to cockpit design and lay- 
out, as well as lighting, communications, training, personnel 
selection and assignment, and the total flight/operation environ- 
ment and the interface between man and machine" (5 ). The 
project was also involved in examining pilot performance in long 
duration simulation studies. 

Past studies of human error accidents did not generally 
evolve from a systematic study of the data. The analyst, in 
acquiring human error statistics, noted the occurrence of these 
accidents more predominantly under certain conditions, places or 
times. It was noted, for example, that a large proportion of 
accidents occurred during the landing phase of flight. This 
prompted interest in measuring the distance of the crash from the 
end of the runway; hence, the definition of the problem of under- 
shooting the runway. As a result of these studies, a host of new 



problems were identified: visibility, judgement, procedures 

training, and engineering requirements. The importance of this 
problem-oriented approach is not questioned. However, if human 
error research planning is to insure that all research require- 
ments are identified, some systematic methodology must be utilized. 

A piecemeal attack on human error has failed to provide the 
perspective needed to approach the problem in an organized and 
systematic manner. A synthesis of the various bits of data accu- 
mulated by the studies done to date could only provide a dis- 
jointed and confusing result, since these • studies have for the 
most part been performed for the purpose of solving a particular 
problem at a particular point in time. A new approach must be 
taken to reclaim the hidden human error information in accident 
investigation records which is not currently shown by statistical 
studies of accidents. The systems approach evolved primarily 
from or along with the research and development of aeronautical 
and space systems. Integral to this systems approach was the 
consideration of safety. 

The investigators have conducted a human factors system 
analysis evaluation of certain NTSB basic accident investigation 
records in order to construct a human error accident model which 
allows graphic demonstration of critical event information. This 
information can be utilized to establish future human factors 
research programs in aeronautics. Analysis of the data also pro- 
vides a basis for evaluating the adequacy of the reporting and 
data-gathering format currently used in the human factors portion 
of accident investigation. 

With the close cooperation of the NTSB Bureau of Aviation 
Safety, the Department of Aerospace and Environmental Medicine 
of the Lovelace Foundation for Medical Education and Research 
has employed a systematic methodology to examine the character- 
istics of the flight prior to the accident (environmental factors, 
aircraft systems, pilot performance, facilities, policies and 
procedures) as well as the contingencies pertaining at the time 
of the accident, to afford consideration of all the elements 
interfacing and interacting in the operation of a complex aero- 
nautical system. The multidisciplinary research approach has 
avoided the negative concept of "accident" traditionally used 
(an unforeseen, unanticipated sudden event that could not be 
helped) . An aircraft accident has been viewed as an eminently 
successful event that is inevitable when preceeded by a series of 
one or more accident-enabling factors or conditions which have to 
be present and without which the accident could not occur. This 
approach deemphasizes pilot "fault" and focuses on the actions 
and decisions, both proper and improper, that are carried out, 
as well as those factors that prompt these actions and decisions. 
The concept encompasses all conditions leading to accidents in the 
man-machine-environment complex, including material failures, etc. 
It is this concept that was used in collecting and analyzing the 
critical conditions and decisions data in our study. 
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The aim of the study was to devise and use a new approach 
to reclaim the hidden human error information in accident inves- 
tigation records not currently shown by conventional statistical 
studies of accidents for the purposes of: 

1. establishing a human error accident system model, 

2. performing an in-depth contingency analysis and 

integrating these data within the framework of 
the system model , and 

3. developing a research planning paradigm for the 

system model. 

A major objective of the study was to discover and identify 
multiple elements of commonality occurring in pilot error- 
related accidents. Figure 1 graphically presents the analytic 
model developed for this study. 


ANALYSIS OP PILOT ERROR- 
RELATED AIRCRAFT ACCIDENTS 


Information 
Source : 



Data 

Refinement: 





Figure 1. 




METHODOLOGY 


Air carrier accident investigations are thorough and 
recognized authorities in their respective fields from both 
industry and government are used as members of the investigation 
team. As a result , minute details about many factors impinging 
upon the ^accident are reflected in the reports. It was there- 
fore felt that the NTSB records on air carrier accident investi- 
gations represented the most thorough and complete sources of 
accident data available to us , with the highest level of expert 
input. 


With the close cooperation of the NTSB Bureau of Aviation 
Safety, the Lovelace Foundation research team subjected certain 
basic NTSB accident records to a human factors/systems analysis 
evaluation. The research approach was multidisciplinary with a 
team whose cumulative experience consisted of a broad background 
in aircraft accident investigation, airline flying, human factors 
engineering, aviation medicine, mathematics, logic and cluster 
analysis and psychology. 

Selected for analysis are accidents with assigned causes 
involving human error for U.S. air carrier turbojet and turbofan 
aircraft from 1958 through 1970. In selecting the accidents for 
this retrospective study, the existing NTSB definition of the 
cause factor classification of human error was neither questioned 
nor challenged. We were not concerned with the primary cause 
factor per se nor the findings of the accident investigation 
board, nor were we interested in identifying a particular 
accident by number, name or location. The primary cause factor 
was used only for the purpose of selecting the basic human 
error accident records for analysis. We accepted the NTSB 
definition of the set. 

Over 200 pilot error-related accident records were re- 
viewed for U.S. air carrier jet aircraft from 1958 through 1970. 
Selected for final in-depth study were 74 accidents. Criteria 
for selection excluded incident reports relating to minor 
accidents such as ground taxi accidents and clear air turbulence 
passenger injuries, reports of which were too brief to be of 
value in this study. 

Data Collection 


Accident data available to us in the NTSB information 
storage system consisted of the basic accident investigation 
record, including all the factual data, group chairman factual 
reports, hearing proceedings, transcripts from cockpit voice 
recorders, air traffic control recorders, flight recorder 
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tracings, flight crew and witness statements, weather reports, 
and the accident narrative report. The NTSB computer file which 
lists selected information abstracted from the records was also 
utilized. Figure 2 presents the NTSB computer file format in 
which data are stored in both raw and coded form. The information 
collected for analysis in this study was of two major types: 
accident characteristics or variables and critical element infor- 
mation. 

Accident Characteristics and Variables 


Of over 350 possible items of data available on each 
accident in the computer file, 37 were selected as having some 
possible relationship to pilot error. These variables, listed 
in Table 1 as variables 00 through 36, include man, machine and 
environmental data. Also selected for study were certain crew- 
related data not listed in .the computer file but available in 
some of the records. These latter items are listed as variables 
37 through 48 in Table 1. Data appearing in the computer file 
were manually verified, corrected, and in some cases previously 
uncoded information entered. The additional information on crew 
flight time, duty time and rest periods, where available, was 
extracted and listed. Accidents occurring prior to 1962 do not 
appear in the NTSB computer file, and these were manually identi- 
fied, searched and data abstracted. Using this data a matrix was 
constructed consisting of 49 variables on each of 74 accidents. 

Of a total of 3626 cells in the matrix, 583 or 16% list "no data 
available", providing an information content of 84%. To this 
matrix was then added the critical element information described 
in other sections of this report. 

The clustering methodology outlined in Appendix A was 
applied to various portions of the data set. The 74 accidents 
were divided into three groups: " non- training/non-midair " (56 

accidents), "training" (12 accidents), and "midairs" (7 accidents) 
Table 2 lists the variables studied (by matrix column number) , the 
scaling used for each variable, assigned levels of logic, and a 
frequency count of the logic states (by the above-noted three 
accident groups) . Assigned levels of logic range from "1" (high- 
est level) through "5". "Not applicable" was assigned "8" and 
"no data" assigned "9" (the lowest level of logic). 

Figure 3 presents a matrix of data (levels of logic) on 49 
variables of the 56 non-training/non-midair accidents, recorded 
in the order in which the data were collected by the investigators 
The columns (variables) are ordered consecutively 00 through 48 
as listed in Tables 1 and 2. The accidents (rows) were assigned 
a permanent identification number, listed at the left of the 
matrix in such a manner as to preserve anonymity. 

Results of the clustering procedure on the data in this 
group of accidents are shown in Figure 4. The numbers along the 

*One training accident was a midair collision and is listed 
in both the training and midair groups. 
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TABLE 1 


ACCIDENT CHARACTERISTICS AND VARIABLES 


00 

Number of engines 


01 

Time of occurrence 


02 

Type of accident (1st) 


03 

Phase of operation (1st) 


04 

Condition of light 


05 

Type weather conditions 


06 

Type instrument approach 


07 

Airport proximity 


08 

Airport elevation 


09 

Runway composition 


10 

Runway condition 


11 

Runway lighting 


12 

Runway length 


13 

Type of terrain 


14 

Pilots involved 


15 

Total flight time (1st) 


16 

Total flight time (2nd) 


17 

Hours in type (1st) 


18 

Hours in type (2nd) 


19 

Pilot age (1st) 


20 

Pilot age (2nd) 


21 

Pilot at controls 


22 

Sky condition 


23 

Ceiling 


24 

Visibility 


25 

Precipitation 


26 

Obstruction to vision 


27 

Relative wind component 


28 

Temperature 


29 

Wind velocity 


30 

Approach lighting availability 

31 

Pilot time last 24 hours 

(1st) 

32 

Pilot time last 30 days 

(1st) 

33 

Pilot time last 90 days 

(1st) 

34 

Duration of this flight 

(1st) 

35 

On duty time (1st) 


36 

Rest period prior to flight (1st) 

37 

Pilot time last 24 hours 

(2nd) 

38 

Pilot time last 24 hours 

(FE) 

39 

Pilot time last 30 days 

(2nd) 

40 

Pilot time last 30 days 

(FE) 

41 

Pilot time last 90 days 

(2nd) 

42 

Pilot time last 90 days 

(FE) 

43 

Duration of this flight 

(2nd) 

44 

Duration of this flight 

(FE) 

45 

On duty time (2nd) 


46 

On duty time (FE) 


47 

Rest period prior to flight (2nd) 

48 

Rest period prior to flight (FE) 



TABLE 2 


(Page 1 of 4) 

Level of Logic 


Matrix 


Level 

Frequency Count 

Col. 



of 

Non-Trng/ 



No. 

Variable (Characteristic) 

Scale 

Logic 

Midair 

Trng 

Midair 

00 

Number of engines 

Four 

1 

36 

10 

2 


Three 

2 

10 

1 

2 



Two 

3 

10 

1 

3 

01 

Time of occurrence 

2206-0559 hours 

1 

8 

3 

1 



0600-1359 hours 

2 

18 

5 

4 



1400-2159 

3 

29 

4 

2 

02 

Type of accident (First) 

On runway 

1 

28 

2 

- 


Off runway 

2 

24 

4 

- 



Midair/Cruise/Departure 

3 

3 

2 

7 



Engine failure 

4 

1 

4 


03 

Phase of operation (First) 

Landing 

1 

24 

4 

- 


Approach 

2 

14 

4 

3 



Inflight/Departure 

3 

5 

3 

4 



Takeoff to 1st pwr. red. 

4 

11 

1 

— 



Static/Ramp 

5 

2 

- 

— 

04 

Condition of Light 

Night 

1 

30 

3 

1 


Daylight 

2 

26 

9 

6 

OS 

Type weather conditions 

Below minirauras 

1 

1 

- 

- 


IFR 

2 

26 

2 

1 



VFR 

3 

29 

10 

6 

06 

Type of instrument approach 

Non-precision 

1 

1 

- 

2 


Contact/visual 

2 

15 

8 

1 



Precision 

3 

18 

1 

~ 



Not applicable 

4,8 

18 

3 

4 



No data 

9 

4 

~ 


07 

Airport proximity 

On airport 

1 

40 

5 

- 


Under 5 miles 

2 

9 

6 

2 



Over 5 miles 

3 

6 

1 

4 



Not applicable 

8 

1 

- 

1 

08 

Airport elevation (ft. ) 

Over 2000 

1 

3 

2 

- 


101-2000 

2 

10 

4 

— 



Sea level - 100 

3 

34 

4 

1 



Not applicable 

8 

6 

2 

6 



No data 

9 

3 

** 


09 

Runway composition 

Macadam 

1 

25 

3 

1 


Concrete 

2 

18 

5 

“ 



Not applicable 

8 

6 

3 

6 



No data 

9 

7 

1 


10 

Runway condition 

Wet 

1 

19 

2 

- 


Clear 

2 

17 

5 

1 

' 


Other 

3 

6 

3 

2 



Not applicable 

8 

7 

1 

4 



No data 

9 

7 

1 

— 

11 

Runway lighting 

On 

Off 

1 

2 

27 

11 

2 

4 

1 

2 



Not applicable 

e 

6 

X 

4 



No data 

9 

12 

5 


12 

Runway length (ft.) 

Under 6000 

1 

4 

- 

- 


6000 - 8000 

2 

17 

3 

1 



Over 8000 

3 

26 

7 

"* 



Not applicable 

8 

6 

2 

6 



No data 

9 

3 


* 

13 

Type of Terrain 

Level 

1 

13 

7 

2 


(off airport) 

Billy/Rolling/Mountaincus 

2 

7 

1 

*" 


Water 

3 

4 

— 

— 



Other 

4 

1 

— 

1 



Not applicable 

8 

30 

3 

4 



No data 

9 

1 

1 


14 

Pilots involved 

PIC/Dual student 

1 

- 

9 

1 


PIC/Chcck pilot 

2 

1 

1 




PIC/copilot 

3 

55 

2 

6 



TABLE 

2 



(Page 

2 of 

4) 

Col. 




Non-Trng/ 



No. 

Variable (Characteristic) 

Scale 

Logic 

Midair 

Trng 

Midair 

15 

Total flight time (hrs.) 

Under 6000 

1 

1 

1 



(Captain) 

6000 - 15,000 

2 

24 

3 

1 



Over 15,000 

3 

31 

8 

6 

16 

Total flight time (hrs.) 

Under 3500 

1 

14 

3 

3 


(First Officer) 

3500 - 10,000 

2 

19 

5 

- 



Over 10,000 

3 

23 

4 

3 



No data 

9 

- 

- 

1 

17 

Hours in type 

0 - 500 

1 

18 

4 

3 


(Captain) 

501 - 2000 

2 

25 

4 

2 



Over 2000 

3 

12 

4 

2 



No data 

9 

1 

- 

- 

18 

Hours in type 

0 - 500 

1 

26 

8 

5 


(First Officer) 

501 - 2000 

2 

22 

2 

2 



Over 2000 

3 

7 

2 

- 



No data 

9 

1 

- 

- 

19 

Pilot age 

Uncler 4 0 

1 

6 

1 

1 


(Captain) 

40 - 50 

2 

35 

10 

4 



Over 50 

3 

13 

- 

1 



No data 

9 

2 

1 

1 

20 

Pilot age 

Under 35 

1 

19 

3 

3 


(First Officer) 

/5 - 45 

2 

28 

6 

3 



Over 45 

3 

6 

3 

“ 



No data 

9 

3 

- 

1 

21 

Pilot at controls 

First Officer 

1 

22 

2 

2 



Other (incl. both) 

2 

3 

8 

- 



Captain 

3 

29 

2 

5 



No data 

9 

2 

- 

- 

22 

Sky condition 

Broken - Overcast 

1 

31 

3 

4 


Clear - Scattered 

2 

15 

7 

3 



P. obscured - Obscured 

3 

8 

- 

- 



No data 

9 

2 

2 

- 

23 

Ceiling (ft.) 

Under 500 

1 

7 

1 

1 



500 - 2000 

2 

22 

1 

1 



Over 2000 

3 

25 

9 

5 



No data 

9 

2 

1 

- 

24 

Visibility (mi.) 

Eq/Under 1 

1 

9 

- 

- 



Over 1-4 

2 

12 

3 

1 



Eq/Over 5 

3 

34 

8 

5 



No data 

9 

1 

1 

1 

25 

Precipitation 

Yes 

1 

27 

2 

1 



No 

2 

27 

9 

6 



No data 

9 

2 . 

1 

- 

26 

Obstruction to vision 

Fog, haze 

1 

16 

3 

3 



Other 

2 

1 

1 

- 



None 

3 

35 

6 

3 



No data 

9 

4 

2 

1 

27 

Relative wind component 

Tail 

1 

8 

1 

1 



Cross 

2 

11 

2 

— 



Calm 

3 

3 

1 

- 



Head 

4 

21 

4 

2 



Not applicable 

8 

6 

1 

4 



No data 

9 

7 

3 

“ 

28 

Temperature (°F.) 

Over 70 

1 

11 

3 

2 


41 - 70 

2 

13 

3 

3 



0-40 

3 

9 

1 

— 



Not applicable 

8 

4 

- 

1 



No data 

9 

19 

5 

1 


13 
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Col. 

No. 

Variable (Characteristic) 

Scale 

Logic 

Non-Trng/ 

Midair 

Trng 

Midair 

29 

Wind velocity (knots) 

Under 10 

1 

26 

6 

3 



10 - 20 

2 

18 

3 

2 



Over 20 

3 

- 

- 

1 



Not,, applicable 

8 

4 

- 

1 



No data 

9 

8 

3 

“ 

30 

Approach lighting 

Not available 

1 

9 

4 

1 



Aval lable 

2 

16 

1 

1 



Not applicable 

8 

18 

3 

4 



No data 

9 

13 

4 

1 

31 

Pilot time last 24 hrs (hrs) 

Over 6 

1 

5 

1 

- 


(Captain) 

4-6 

2 

22 

2 

4 



Under 4 

3 

27 

7 

2 



No data 

9 

2 

2 

1 

32 

Pilot time last 30 days (hrs) 

Over 70 

1 

10 

1 

1 


(Captain) 

50 - 70 

2 

9 

- 

1 



Under 50 

3 

8 

2 




No data 

9 

29 

9 

5 

33 

Pilot time last 90 days (hrs) 

Over 210 

1 

5 

- 

- 


(Captain) 

150 - 210 

2 

15 

- 

1 



Under 150 

3 

16 

1 

1 



No data 

9 

20 

11 

5 

34 

Duration this flight (hrs) 

Over 6 

1 

2 

- 

- 


(Captain) 

3-6 

2 

19 

5 

1 



Under 3 

3 

35 

4 

6 



No data 

9 

- 

3 

- 

35 

On duty time (hrs) 

Over 7 

1 

14 

- 

2 


(Captain) 

4-7 

2 

26 

6 

3 



Under 4 

3 

13 

3 

1 



No data 

9 

3 

3 

1 

36 

Rest period prior to flight 

Under 12 

1 

4 


_ 


(hrs) (Captain) 

12 - 16 

2 

9 

3 

3 



Over 16 

3 

41 

6 

3 



No data 

9 

2 

3 

1 

37 

Pilot time last 24 hrs (hrs) 

Over 6 

1 

5 

1 

_ 


(First Officer) 

4-6 

2 

22 

2 

4 



Under 4 

3 

25 

4 

2 



No data 

9 

4 

5 

1 

38 

Pilot time last 24 hrs (hrs) 

Over 6 

1 

5 

_ 

_ 


(Second Officer) 

4-6 

2 

13 

2 

3 



Under 4 

3 

20 

4 

- 



No data 

9 

18 

6 

4 

39 

Pilot time last 30 days (hrs) 

Over 70 

1 

7 

_ 

_ 


(First Officer) 

50 - 70 

2 

8 

• 

2 



Under 50 

3 

7 

- 

1 



No data 

9 

34 

12 

4 

40 

Pilot time last 30 days (hrs) 

Under 70 

1 

2 


1 


(Second Officer) 

50 - 70 

2 

10 

- 

- 



Under 50 

3 

2 

1 

- 



No data 

9 

42 

11 

6 

41 

Pilot time last 90 days (hrs) 

Over 210 

1 

7 

«. 

1 


(First Officer) 

150 - 210 

2 

12 


- 



Under 150 

3 

11 

- 

1 



No data 

9 

26 

12 

5 

42 

Pilot time last 90 days (hrs) 

Over 210 

1 

2 

_ 



(Second Officer) 

150 - 210 

2 

5 

1 

- 



Under 150 

3 

10 

- 

- 



No data 

9 

39 

11 

7 

43 

Duration this flight (hrs) 

Over 6 

1 

2 


_ 


(First Officer) 

3-6 

2 

19 

3 

1 



Under 3 

3 

33 

7 

6 



No data 

9 

2 

2 

- 
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Col. 




Non-Trng/ 



No. 

Variable {Characteristic) 

Scale 

Logic 

Midair 

Trng 

Midair 

44 

Duration this flight (hrs) 

Over 6 

1 

2 




(Second Officer) 

3-6 

2 

15 


1 



Under 3 

3 

23 


3 



No data 

9 

16 

5 

3 

45 

On duty time (hrs) 

Over 7 

1 

15 


2 


(First Officer) 

4-7 

2 

26 

6 

3 



Under 4 

3 

12 

3 

1 



No data 

9 

3 

3 

1 

46 

On duty time (hrs) 

Over 7 

1 

11 

1 

1 


(Second Officer) 

4-7 

2 

17 

5 

2 



Under 4 

3 

10 

1 

- 



No data 

9 

18 

5 

4 

47 

Rest period pfior to flight 

Under 12 

1 

4 

1 

_ 


(hrs) (First Officer) 

12 - 16 

2 

11 


3 



Over 16 

3 

37 

7 

2 



No data 

9 

4 

4 

2 

48 

Rest period prior to flight 

Under 12 

1 

4 

_ 

_ 


(hrs) (Second Officer) 

12-16 

2 

7 

1 

2 



Over 16 

3 

27 

5 

1 



No data 

9 

18 

6 

4 


15 



CHARACTERISTICS 


ACCIDENTS 


0000000000 1 111111111222222222233333333334444*4444 
0123456769012345678901234567890123456789012345678 


00 1 12112231321 13832311213321111922323223332323292221 

002 1322232132223833222321233234111223323223222332233 

003 1222232132223833221223223232111333223333939222233 

005 1322232231222133311323233234921219333229999333333 

006 1311122131123833211323121131129219212223299221122 

007 12112391318938 33 32122123 32348U39933 3339999333333 

008 1311123131312832311223112112992299311229999331111 

009 1311123132112333331313132134991222223223239222233 

010 1222232122992833232223133233999323333239913331323 

OH 1111123131113332221233112211212199113119999111133 

012 1321232131292833232223233234119292312239939331122 

013 121 12321 31 222933221321 13 3234 22 133"2333332293 333333 

014 2322132211993133311223133234212399333239999332323 

015 1311123132213833232321123134322311223331222222233 

016 1223239331992233311229223232999312323392929392939 

017 1311123132112833223323322124322323223322233222232 

018 1311123132213833311211112112222339223339999222233 

019 1222232132292833332311233234129119212112299221122 

020 2322122131112832111213112112922299213229999221133 

022 1311132121213833111211233233222993393339933333333 

023 1211233132292832199211233234819399113339999111133 

026 1322123221113232121213322112911322222392999222929 

030 3111123131112132121991321134322392333399912393939 

031 2222223131193132122213121114922293223299919292939 

032 2222223231912432111123921211212233322992999999999 

033 1221233139323132223131123234119333323333233332933 

035 1311133131112232322211133131112112313191929391939 

037 3311232131121132312211123134912293313299939391939 

040 3311239132323132321191233134919291223299939292939 

04 1 322211323231 1133321221331211219299312299999391929 

042 1222223299119332222339123191992211312299999391929 

043 3321 13229931923112211323323491 1399333399999393939 

046 221 12331 1 1993832111 1 13133134229393223319999222132 

049 1314224368088833312223133138288233323221113332233 

050 1114134131 193833332321123939228292212229923221122 

05 1 1314134 122212633132211 133232318393323339922332233 

053 1115124122113832222223122119318399323339999332233 

054 2124124122213632121223123132328223311222232331111 

056 1224124221313133333221332213318339333333399333333 

057 2323124388888332323213222118218393333339923333333 

060 2315134121113832122113233239118199321119999332211 

064 2114134132111832322221123219 1 10399323339999332233 

067 1324234132223833233231133232213111312111111331122 

069 1311229139113833222323122199999312293999999229933 

, 070 1133128888888832211231199998998399223339999222233 

071 1211223139212333222321111299999299223229999222233 

078 332212830888822321122331 1138888393333399939393939 

080 111413813111383312122293313221839-2333339921333333 

063 3344234131993132122212233239928392323339929332233 

084 3322121299999233332213123112912322323391929392939 

085 1324124121113132233232113114318211223221219222233 

055 3224124122212133311213122114328222331293929393919 

074 121 12321 19223833311993133231 119992339999933333399 

079 39331343888882333222231232389082322232919 1 “29 29 39 

086 231 1232131221 1322 12223233234 121 29939 399999» 999999 

092 133323438888883333332139H3B888299329229999332299 


Figure 3. Data on 56 Non-Training/Non-Midair accidents 
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91132311113019922321992229221113333992233 
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015 

linui. 

mum 

linin' 
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Figure 4. Clustered data on 56 non-training/non-midair accidents. 
Lines demarcate major clusters and subclusters. 
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upper border of the matrix identifying the variables (00 through 
48) correspond to those listed in Tables 1 and 2. In studying 
a clustered matrix, patterns and interrelationships are more 
easily visualized if each level of logic is assigned a color 
which is then superimposed on each matrix cell. As a result of 
the nature of clustering methodology certain patterns of data 
migration occur within the matrix. Accidents and variables having 
high information content drift toward the upper left portion of 
the matrix, while variables and accidents having relatively low 
information content drift toward the lower right of the matrix. 
Furthermore, the scaling to which a variable is subjected before 
matrix entry (and the resultant frequency of levels of logic 
within that variable) assumes importance in terms of the cluster- 
ing results. The decisions regarding scaling of the variables 
listed in Table 2 were reached based on the experience of the 
research team. This matrix can be used as an investigative 
model to examine the effects on clusters and patterns of manipu- 
lating variables (rescaling, combinations of variables, ratios, 
etc. ) . 


The clustered matrix of the 56 accidents in Figure 4 points 
out a number of interesting results and commonly shared traits. 

The 56 accidents are divided into three distinct clusters on the 
basis of airport proximity (the first column, 07) . Forty acci- 
dents occurred on the airport, nine at less than five miles and 
seven over five miles. The cluster of 40 on-airport accidents 
is further divided into three subclusters on the basis of "number 
of engines" (column or variable 00) - over one half of the on- 
airport accidents involved four-engine aircraft. Eighteen acci- 
dents are identified as having occurred on the airport, in four- 
engine aircraft, in~broken to overcast sky conditions. Of these 
latter 18, 11 occurred at night and seven of these were associated 
with precipitation. The foregoing is an example of the kinds of 
observations possible using this technique. One other interesting 
finding in this matrix which is corroborated in Table 2 is that 
within this group of 56 accidents, 41 list the captain as having 
had over 16 hours of rest prior to that flight. Simple observa- 
tion of the "no data" points within the matrix (listed as "9") 
quickly demonstrates the magnitude of missing data and areas 
requiring standardization of accident data collection. Many hours 
have been spent in studying these clusters and many patterns 
emerge despite the small number of accidents. This technique is 
a useful tool in studying large data sets of aircraft accident 
information. 

The clustered data on the twelve training accidents is 
shown in Figure 5. Although the total number of accidents is 
relatively small, a number of commonly shared characteristics 
are visible. All 12 accidents were actual training (or profi- 
ciency) flights. Of the ten involving four-engine aircraft, six 
had a low experienced first officer (0-500 hours in type) . All 
but one of the 12 occurred on or within five miles of the airport 
(column 07) . Nine of the 12 occurred during daylight hours 
(column 04) . Ten occurred in VFR conditions (column 05) . The 
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captain was between 40 and 50 years of age in ten of the acci- 
dents. These data must be related to the characteristics of 
the general population of routine training flights. 
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Figure 5., 


The clustered data on the seven midair accidents are depicted 
in Figure 6. Due to the limited number of accidents the cluster- 
ing procedure by itself yields little or no new information. 
However, it is evident from the matrix data that six of the acci- 
dents occurred in daylight under VFR conditions. All but one had 
been in flight under three hours. Six of the seven captains 
involved had over 15,000 hours total flight time but three of the 
seven had relatively little time in type (0-500 hours) . Five of 
the seven first officers had low time in type (0-500 hours). 

Four of the flights had a 3-man crew; the other three had a 2-man 
crew. 
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Figure 6. 
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Determinative or Identification Schemes 


Another aspect of this clustering methodology is the 
ability to determine, when the next event occurs, what the 
shortest route would be to identify that event as being similar 
to or different from previous events. This requires a method 
for constructing identification schemes from accumulated data. 

The variables to select in optimally identifying the new event 
are those variables which, on the basis of past experience, 
separate or partition previous events at the most efficient rate. 

The theoretical aspects of this separating or partitioning 
of events are discussed in Appendix B and applied to the critical 
element data on page 40. 
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ACCIDENT CRITICAL ELEMENTS 


In the critical element portion of the data collection, 
each of the selected accident records was reviewed by the team 
members. From these records were extracted critical conditions 
and decisions. Critical elements were considered to be those 
causal elements, attributes, events or conditions which signifi- 
cantly contributed or were related to the occurrence of the 
accident. These critical elements were broadly divided into 
"critical conditions" and "critical decisions" and consisted of 
subjective, qualitative statements whose subject and predicate 
were then listed for each accident. The critical elements thus 
identified and previously submerged in the records were collected 
in a format amenable to meaningful analysis both by themselves 
and in combination with the accident variables previously 
mentioned. For purposes of matrix entry and analysis, the 19 
critical elements were transformed into binary data (element 
present or absent, yes or no) . The elements were also studied 
in subject/predicate form and results and discussion of these 
data follow. 

Critical Conditions 


The critical condition category model is depicted in 
Table 3. 

Table 4 depicts the type of accident by phase of opera- 
tion. Classification and phase of operation are as coded by 
the NTSB . Those operations that are concerned with reaching 
the runway touchdown point have the highest frequency of occur- 
rence in the accident population, i.e. 13 undershoots, nine 
hard landings, eight overshoots, and seven ground-loop/ swerves. 
Forty-seven out of the 74 accidents (64%) occurred during the 
landing phase of operation. 

Table 5 depicts critical conditions in 56 non-training/ 
non-midair accidents, seven midair accidents and 12 training 
accidents. On the ordinate are listed the ten critical condi- 
tions and on the abscissa are listed the code numbers for each 
accident in its respective grouping. The parenthetical number 
shown with each critical condition (e.g. 38 following "experi- 
ence") indicates that there were 38 accidents in which experience 
was a critical condition within the accident set. 

Of the 12 training accidents, one accident was actually an 
on-line training event. The 11 others were training or pro- 
ficiency type flights. In the training group, there were eight 
machine-related accidents, six decision-related accidents and 



TABLE 3 


CRITICAL CONDITION CATEGORIES 


1. Experience 

a. Low pilot time in type 

b. Low copilot time in type 

c. Low pilot time in position (as Captain) 

d. Low copilot time (total) 

e. Other (includes: recent experience, training, flight 

engineer, age differences, student pilot new, crew new, 
student pilot dull, new airport) 

2. Distraction 

a. Communications or traffic (excessive communications with 

ATC or looking out for traffic) 

b. Confusion (last minute approach changes or other confusion) 

c. Hurry (close departure on same runway or other hurrying) 

d. Holding or delay 

e. Other (includes: wake turbulence, numerous distractions, 

foreign student, first officer monitoring instruments, 
interrupted checklist, fuel burn, paperwork, poor destina- 
tion weather, instructor pilot checklist, takeoff position 
holding, ashtray fire) 

3. Crew Coordination 

a. Disagreement (disagreement on approach or configuration, or 

other pilot calls "off profile") 

b. Jumpseat occupant or other additional crew 

c. No required altitude callouts 

d. Pilot acting as instructor 

e. Other (includes: loose student/instructor relationship or 

other interactions such as flaps without student knowing/ 
altitude confusion, distrust first officer, thought con- 
tinuing takeoff, gear up without visual verification, both 
pilots on controls, non-compliance, confusion on who was 
flying) 

4. Neglect 

a. No cross-check on ILS 

b. Improper use of checklist 

c. Improper rest/procedure 

d. Other (includes: company did not revise checklist, other 

aircraft collision light off, ATC, Mach trim switch, 
engine reversing indicator lights, VOR out, clearance 
deviation) 


5. Air Traffic Control 

a. Delayed landing clearance 

b. Confusing radar vector 

c. Advised of traffic 

d. Poor, weak or malfunctioning radar or radar return 

e. Other (includes: no acknowledgement, no advisories, vector 

confirmation, advisory holds) 
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TABLE 3 

CRITICAL CONDITION CATEGORIES (continued) 


6. "Decisions" 

a. Off acceptable profile 

b. Institutional decisions: OK to operate 

c. Copilot flying, taken over by pilot 

7. Work/Rest (fatigue) 

a. On duty over 8 hours 

b. Minimum rest 

c. Early morning departure 

8. Machine 

a. Gross weight (overweight or heavy gross weight) 

b. Simulated engine shutoff (engine failure simulation) 

c. System failure 

d. Other (includes: simulated rudder loss, flight director 

oscillation, spoiler deployment and retraction, battery 
switch, 3 & 4 engine reverse, slow spool, air noise, 
parking brake versus mechanical failure, seat failure) 

9. Airport 

a. Stopping problem (runway slippery, wet, slush, braking 

action poor, or tire residue) 

b. Touchdown problem (runway short or displaced threshold) 

c. Vertical guidance problem (no approach light, approach 

lights out, or localizer only) 

d. Runway hazards (upslope threshold, exposed lip or dropoff) 

e. Other (includes: runway markings obliterated, uncontrolled 

airport, irregular lights, loose pavement, hilly terrain) 

10. Weather 

a. Visibility problem (heavy rain at threshold, below circling 

minimums, fog, snow or haze or other visibility restric- 
tions) 

b. Thunderstorm influencing airport or enroute weather 

c. Wind gusty 

d. Other (includes: same route, weather above circling mini- 

mums, enroute weather, freezing drizzle, venturi wind) 



TABLE 4 


Type of Accident 
NTSB Code/Description 


Phase of Operation 

Taxi Takeoff Inflight Landm 


A Ground Loop/Swerve 
B Dragged wingtip 
C Wheels up 
E Gear collapsed 
G Hard landing 


J Overshoot 

K Undershoot 

LO Collision with A/C 
(both inflight) 

Ml Collision with ground/ 
water (uncontrolled) 

MO Collision with ground/ 
water (controlled) 

Nl Collided with trees 

N6 Collided with runway/ 
approach lights 

NB Collided with ditches 

0 Hail damage 

Q Stall 

SO Airframe failure 
(inflight) 

U Engine failure/ 
malfunction 


3 uncontrolled altitude 
deviation 
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TABLE 5 

CRITICAL CONDITIONS 
Non-Trainin^/Non-Midair (56) 


Midair (7) 
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five experience-related accidents. Excluding the on-line train- 
ing accident, the significant observation is that seven of the 
11 training flights were operating with simulated engine failures. 
In most cases of operating with simulated engine failures, pro- 
files of flight were unacceptable (decision information) . Flight 
parameters were allowed to continue beyond the point where the 
instructor could regain control of the aircraft before the acci- 
dent. Three out of the five accidents where experience was 
involved included an instructor pilot who was new at his job. 

Two of the crew coordination items out of the four were inter- 
ruption of cockpit routine by a jumpseat occupant or additional 
crew member. The one item of neglect was an instance where the 
company did not update the emergency checklist. In general the 
training accidents involve conditions of relatively good weather 
and relatively few airport problems. 

Of the seven midair accidents, one was also a training 
accident. In this event set of midair accidents, both distrac- 
tions and ATC problems predominate. In fact, each accident has 
either a distraction or an ATC condition or both. Distractions 
included "hurryup" approaches, evaluating runway identifier 
lights, an ashtray fire and a radio frequency change. Of the 
four ATC conditions, two involved poor or malfunctioning ATC 
radar equipment, and two involved situations in which no traffic 
advice or warnings were given. If one ATC problem was shifted 
to neglect, the seven midair accidents would then be: two acci- 

dents where ATC radar was malfunctioning or of inadequate quality; 
one where ATC did not advise or warn of traffic; one where the 
air carrier aircraft exceeded its clearance limit; and three 
where neglect on the part of the aircraft other than the air 
carrier aircraft played a vital role in the accident (such as 
not reporting position, not using navigation lights, or deviat- 
ing from a clearance) . 

Some general observations may be made about the 56 non- 
training/non-midair accidents. The most commonly observed 
characteristic among the 56 accidents is experience, which 
occurs in 38 accidents. Second in occurrence is airport condi- 
tions which appear in 32 accidents. Decisions appeared in 31 
accidents, and weather in 28. Further partitioning of the data 
is possible. For example, in the experience grouping, there 
were 28 accidents with copilot low in time in type. Twenty 
accidents occurred with low pilot time in type. Sixteen acci- 
dents occurred where both the pilot and copilot had low time in 
type. Again, low time in type is described as less than 500 
hours in the particular type of airplane involved. Partitioning 
the condition of airport, we find 16 accidents where there was 
a stopping problem, nine accidents where there was a touchdown 
problem and seven accidents where there was a vertical guidance 
problem. In the decision classification, the most frequent 
characteristic appearing was "off profile", occurring in 21 
accidents. The predominant weather condition was visibility, 
occurring in 20 accidents. 
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In an attempt to more closely study certain subcharacteris- 
tics (e.g. subcategories of decisions, such as 6a, off accept- 
able profile) , a screening technique was applied to the data. 

The technique is graphically presented in Figure 7. 

Of 21 off profile accidents (first screen) : 

Seven were experience only related, with one or 
both pilots having low time in type; 

Four involved changeover of control from the 
copilot to pilot (In only one of these cases 
was there an experience-related problem; three 
of the four had weather problems, and of these 
three, two had airport problems) ; 

Four shared a common characteristic of a dis- 
agreement of aircraft operation. In three of 
these, the pilot in command was also acting as 
instructor pilot. Two were also related to 
experience, with both pilot and copilot having 
low time in type ; 

Three experience-related accidents had no 
attending weather or airport problems, but all 
had copilot low time in type; 

The remaining three accidents showed airport 
problems, with two being touchdown problems 
and one a vertical guidance problem. 

Of 16 accidents with low copilot time in type (second 
screen) : 

No off profile condition noted; 

Nine associated with weather problems (seven 
with poor visibility, two with thunderstorms) ; 

Four were machine-related, with such problems 
as antiskid device inoperative, stall warn- 
ing device inoperative, etc.; 

Three shared no common trait other than low 
copilot time in type, but all three were 
accidents in which information was sparse; 

In the total group of 74 accidents, seven had 
some type of system malfunction; six of these 
occurred in this group. 

Of 14 accidents with airport-related problems: 

Neither off profile nor low time in type for 
copilot were present; 














Seven had stopping problems; 

Four had runway hazards; 

Two had touchdown problems; 

One had a vertical guidance problem? 

Eight contained elements of distraction, 
with five accidents having holding or 
other delays. 

The remaining five accidents (fourth screen) do not have 
the above characteristics in common. Further observation of the 
data on these accidents, however, reveals other interesting facts 
For example, none were in the approach and landing phase of opera 
tion? all captains had over 6000 hours total flying time, were 
well rested, and had not flown in excess of 210 hours in the pre- 
ceding 90 days; pilots had had less than six hours flying time in 
the preceding 24 hours? and all accidents occurred between 1400 
and 2200 hours, local time. 

The 56 non-training/non-midair accidents are baffling and 
complex in nature. Another technique was devised to study this 
group. This method was opposite in nature to previous methods. 
Those characteristics found in the largest number dealt with 
experience, decisions, airport and weather problems. In reverse, 
we withdrew those accidents . which showed none of the most common 
traits. Seven such accidents were of this type. The most unique 
trait observed within this group of seven was that they contained 
the five atypicals described above. In an examination of the 
seven accidents it was found that each accident was so unique 
that a sibling could not be found in the accident group of 74. 
While some of these seven accidents were complex, it could be 
observed that if one condition in the enabling factors could be 
changed, an accident might not have happened. Due to the sensi- 
tive nature of the accident data, specific description of 
enabling factors cannot be mentioned. In general the "one of a 
kind" atypical accident enabling factor falls into such groups 
as design hazards, physical hazards, and system malfunctions. 

It may be that there is such a thing as an "irreducible minimum” 
of human factors accidents and that these atypical or "freak" 
accidents may represent that minimum. Certainly we cannot sub- 
scribe to the ludicrous concept that there is an acceptable 
tolerance level for human factors accidents, but we do feel that 
there indeed may be a small group of accidents that will not be 
classifiable as reasonably preventable; and zero human error, 
while a superb ideal, remains a utopian goal. 

There were 11 accidents in which only one of the common 
traits could be found. This group will be the subject for 
further examination. It presently appears to be related to the 
previously discussed seven (none of which contained a common 
trait) but in less subtle enabling factors. 



The remaining group of 37 accidents possesses two or more 
of the common traits. Again, reviewing the common traits: 
experience, i.e. low pilot time in type, low copilot time in 
type; decisions, i.e. off profile flight conditions; airport, 
i.e. stopping problem, touchdown problem, vertical guidance 
problem; weather, i.e. visibility problem. It appears to this 
group of investigators that methods can be devised to limit or 
provide warning in the event that two or more of these enabling 
factors are present. If such a system could be devised and 
placed in use, it should be the foundation of an effective 
accident prevention program. 


Critical Decisions 


The purpose of this portion of the study was to examine in 
detail those pilot decisions that were made in the period just 
prior to the accident event. This data set was collected 
separately from the critical conditions using a somewhat differ- 
ent approach. Consequently, the "decisions' 1 listed in the 
critical condition classification are distinct from the decisions 
in this section. 

Inappropriate decisions preceding a pilot error-related 
aircraft accident are often the result of inaccurate perceptions 
or of failure to make timely decisions (19). In order to analyze 
critical decisions collected on 73 accidents (one of the selected 
74 accidents had no available information from which to abstract 
critical decision elements) , a listing in the form of one- 
sentence descriptors was made for each accident following review 
of the record. Inter-investigator variability was checked by 
having each record analyzed independently by several analysts. 
This cross-check showed no meaningful differences between inves- 
tigators. 

The working definition of a critical decision adopted for 
data collection considered it to be a critical information pro- 
cessing function (choice) of a flight crew member occurring in 
the time period just prior to the accident event. Identification 
of critical decisions was cued to the review of factual accident 
records for control inputs, verbal communications, post-accident 
flight crew statements, flight profile, etc. When all the 
decision data had been collected, Altman's classification scheme 
was adapted in order to catalog and classify the data set ( 3 ) . 
Table 6 lists the major categories of critical decisions in the 
73 accidents. 

The critical decisions listed for all the accidents were 
then sorted by accident into three major groups: Non-training/ 

non-midair (55 accidents) , training (12 accidents) , and midairs 
(7 accidents) . One training accident was a midair collision; 
thus both the training and midair groups share one accident (a 
total of 18 accidents) . 
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TABLE 6 


CRITICAL DECISION CATEGORIES 


1. Decisions resulting from out-of-tolerance (off profile) 
conditions 

a. Takeover of controls 

b. Verbal instructions between pilots 

c. Excessive deviation called out 

d. Inadequate braking observed 

e. Assistance in flight control operation 

f . Attempt to regain directional control 

g. Go- around initiated 

h. Other* 

2. Decisions based on erroneous sensory inputs 

a. Approach continued visually 

b. Decided profile within limits 

c. Misleading cockpit display 

d. Misleading navigation information 

e. Runway/braking misinformation 

f. Final approach/flare profile misinformation 

g. Standard operating procedure distraction 

h. Other 

3. Decisions delayed 

a. Takeover of flight controls or assistance 
b* Go- around decision 

c. Takeoff abort 

d. Thrust lever movement 
e* Other 

4. Decision process biased by necessity to make destination or 
press-on (meet schedule) 

a. Continued flight with equipment failure 

b. Altered cockpit procedures 

c. Continue with weather conditions deteriorating 

d. Runway misinformation 

e. Decision involved approach procedure 

f . Other 

5. Incorrect weighting of sensory inputs or responses to a 
contingency 

a. Deviation from checklist/altitude callouts 

b. Icing of aircraft 

c. Disregard of cockpit displays 
d* Traffic information disregarded 

e. Disregard information on landing environment or conditions 

f. Safety degradation due to training 

g. Other 
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TABLE 6 

CRITICAL DECISION CATEGORIES (continued) 


6. Incorrect choice of two alternatives based on available 
information 

a. Left cockpit 

b. Landed runway with unfavorable conditions 

c. Flew visual approach 

d. Other 

7. Correct decision 

a. Checked approach light level 

b. Confirmed minimums 

c. Took over and flew approach 

d. Other 

8. Overloaded or rushed situation for making decision 

a. Primary attention diverted 

b. Aircraft power difficulty 

c. Observed traffic and rolled aircraft 

d. Other 

9. Desperation or self-preservation dec.ision 

a. Directional control or stopping problem 

b. Airborne loss of control 

c. Avoid ground contact 

d. Avoid other aircraft 

e. Other 


* Critical decisions listed as "other" in each category were 
miscellaneous and too few in number to list herein. 
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Non-Training/Non-Midair Accidents . Table 7 summarizes the 
critical decisions listed for the 55 non-training/non-midair acci- 
dents. (Entries are not mutually exclusive; one accident may have 
critical decisions listed under several categories.) Within this 
group of 55 accidents, the greatest number of critical decisions, 
37, were made as a result of conditions being out of tolerance 
with respect to a normal flight profile. A pilot often accepts 
an off-profile deviation to accommodate air traffic control con- 
straints, airport noise regulations, passenger comfort and other 
restrictions. The second highest number of critical decisions, 

30, in this group were those that were based on erroneous sensory 
inputs. Leak observed this phenomenon in a study of USAF approach 
and landing accidents: "Although a number of areas need increased 

emphasis, the critical one appears to be information." (11) Willis 
and Bryant refer to the false hypothesis concept where the more 
limited or the more vague the information, the less time the pilot 
has for evaluation (23) . 

The next group of decisions, 27, involves incorrect weight- 
ing of sensory inputs or responses to a contingency. Scucchi 
and Sells refer to this hazard of inaccurate information process- 
ing as one that flight training and flight operations must come 
to recognize (19) . Those decisions which are biased by pressure 
to make the destination or to meet the schedule account for the 
next highest number of critical decisions, 26. This kind of 
decision may be a result of conditioning pilots receive every 
day of continuing the planned flight profile. There seems to be 
a "continue on and it will work out" attitude in accidents involv- 
ing this type of decision. 

Training Accidents . Table 8 summarizes the critical 
decisions made in 12 training type aircraft accidents. The two 
most common critical decision categories in this group are those 
decisions delayed beyond the time required and those made as a 
result of conditions out of tolerance, eight. "Delaying decisions" 
is inherent in pilot training procedures employed by flight in- 
structors. The tendency is to allow the student as much time as 
possible to recognize and correct a dangerous situation before 
taking over. 

Midair Collision Accidents . In the midair collision acci- 
dents the most frequent decision listed is that based on the 
requirement to continue to the destination or make schedule, 
five. Decisions based upon incorrect weighting of inputs numbered 
three. An example of this kind of decision is one where the 
pilot is issued air traffic information while other cockpit 
duties require' his primary concentration. Table 9 summarizes 
the decisions made in the seven midair collisions. 

Combinations of Critical Decisions . The (critical decision 
data were also studied in terms of the combinations of decisions 
appearing within a given accident. Table 10 lists the frequency 
of more common combinations of two and three decisions found 
within the 55 non-training/non-midair accidents. Whenever 



TABLE 7 


SUMMARY OF CRITICAL DECISIONS 
IN 55 NON-TRAINING/NON-MIDAIR ACCIDENTS 


TYPE OF DECISION 


NO. OF DECISIONS 


1. Conditions out of tolerance 37 

2. Erroneous inputs 30 

5. Incorrect weighting of inputs 27 

4. Make destination/Press-on 26 

3. Delayed 20 

9. Desperation 10 

8. Overloaded or rushed 9 

7. Correct decision 8 

6. Incorrect choice 5 
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TABLE 8 


SUMMARY OF CRITICAL DECISIONS 
IN 12 TRAINING ACCIDENTS 


TYPE OF DECISION 


NO * OF DECISIONS 


3. Delayed 8 

1. Conditions out of tolerance 8 

5. Incorrect weighting of inputs 7 

2. Erroneous inputs 3 

9. Desperation 3 

8. Overloaded or rushed 2 

4. Make destination or Press-on 1 

6. Incorrect choice 2 

7 . Correct decision 
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TABLE 9 


SUMMARY OF CRITICAL DECISIONS 
IN 7 MIDAIR COLLISIONS 


TYPE OF DECISION 


NO. OF DECISIONS 


4. Make destination or press-on 

5. Incorrect weighting of inputs 

1. Conditions out of tolerance 

2. Erroneous inputs 
9. Desperation 

8. Overloaded or rushed 

3 . Delayed 

6. Incorrect choice 


5 

3 

2 

2 

2 

1 


7, 


Correct decision 



TABLE 10 


COMBINATIONS OF CRITICAL DECISIONS 
IN NON-TRAINING/MIDAIR ACCIDENTS (55) 

COMBINATION OF 2 DECISION CATEGORIES NO. OF ACCIDENTS 


Conditions out of tolerance 
and Erroneous inputs 

Conditions out of tolerance 

and Make destination or press-on 

Conditions out of tolerance 

and Incorrect weighting of inputs 

Conditions out of tolerance 
and Delayed j 

Erroneous inputs I 

and Make destination or press-on 

Erroneous inputs / 

and Incorrect weighting of inputs 

Delayed j 

and Incorrect weighting of inputs 

Make destination / 

and Incorrect weighting of inputs 


COMBINATION OF 3 DECISION CATEGORIES 


Conditions out of ■ 

tolerance , 

Delayed 

1 

and Incorrect weighting of inputs 

Conditions out of 

tolerance, 

Erroneous inputs 


and Make destination or press-on 

i 

Conditions out of ' 

tolerance. 


Erroneous inputs j 
and Delayed 

Conditions out of tolerance, 
Erroneous inputs 

and Incorrect weighting of inputs 

Conditions out of tolerance, 

Delayed 

and Make destination or press-on 

Conditions out of tolerance, 

Make destination or press-on 
and Incorrect weighting of inputs 


18 

17 

16 

16 

13 

13 

11 

11 



8 


6 


6 


6 


37 



sequential decisions of the categories listed in this table were 
made, they often resulted in inaccurate information processing. 
There appears to be a need for further research into more effic- 
ient means for a pilot to make decisions involving conditions 
out of tolerance, incorrect weighting of inputs, erroneous 
sensory inputs and combinations thereof. 

Associations of Critical Decisions with Critical Conditions 


The following chart lists the number and type of critical 
decisions found to appear concurrently within an accident, with 
specific critical conditions: 

1. Condition: Off acceptable profile (21 accidents) 

Associated Decisions No. of Decisions 

1. Resulting from "out of tolerance" 23 

2. Erroneous inputs 12 

4. Make destination/press-on 12 

5. Incorrect weighting of inputs 12 

3. Delayed 9 

7. Correct decision 3 

8 . Overloaded 1 

9 . Desperation 1 

6. Incorrect 

2. Condition: Low copilot time in type (16 accidents) 

Associated Decisions No. of Decisions 


1. Resulting from "out of tolerance" 

4. Make destination/press-on 

2. Erroneous inputs 

3. Delayed 

5. Incorrect weighting of inputs 
9. Desperation 

6. Incorrect 
8. Overloaded 

7. Correct decision 


11 

10 

9 

6 

5 

5 

3 

2 

1 


38 



3. Condition: Airport (14 accidents) 


Associated Decisions 

5. Incorrect weighting of inputs 

1. Resulting from "out of tolerance" 

2. Erroneous inputs 

3. Delayed 

4. Make destination/press-on 

7. Correct decision 

8. Overloaded 

6. Incorrect 

9 . Desperation 


Ho. of Decisions 
11 
10 
7 
6 
5 
4 
3 
1 


It should be noted that with each of the three critical 
conditions listed above, the five most frequently associated 
critical decisions were (1) decision regarding conditions out of 
tolerance, (2) decisions from erroneous inputs, (3) delayed 
decisions, (4) make destination/press-on based decisions, and 
(5) decisions based on incorrect weighting of inputs. "Airport 
problem" accidents most often involve erroneous decisions due to 
incorrect weighting of sensory inputs. Accidents associated with 
low copilot time in type and off-profile are often associated 
with decisions regarding conditions out of tolerance. 



CLUSTER ANALYSIS OF CRITICAL ELEMENTS 


The clustering method using truth table classification 
(Appendix A) was applied to the critical conditions and decisions 
data available for the 56 non-training/non-midair accidents. The 
input data matrix consisted of 19 critical elements (in binary 
form, "present" or "not listed") on 56 accidents. Results of the 
clustering procedure are depicted in Figure 8. One can see in 
the matrix how this agglomerative type of clustering links, by 
the state of a variable (in this case a given critical element 
present or not listed) , the 56 accidents and creates progressively 
smaller subclusters. Also evident are groups of accidents sharing 
multiple critical element states. 

Using this clustered matrix of critical elements the deter- 
minative or identification scheme (Appendix B) was applied. The 
partitioning or separation ability of each critical element was 
calculated and these are listed in Table 11. Critical element 10 
separates 784 of the 1540 pairs. Critical elements 10 and 15 
separate 1175 of the 1540 pairs. The first nine elements listed 
in the Table separate 1538 (99.9%) of the 1540 possible pairs of 

accidents these then are the "nine best tests" that, applied 

in sequence, will provide optimal separation of events. This 
information is graphically presented in Figure 9 and compares 
the resultant curve of separation with the theoretical curve (in 
which the nine best tests would separate 100% of the pairs) . 

These selected nine critical elements were then used to 
construct a matrix of the 56 accidents and the data were clus- 
tered using truth table classification. Figure 10 shows the 
clustered results and demonstrates how the nine critical elements, 
when applied in sequence listed, progressively partition the 
56 events optimally. For example, the critical element "weather" 
(number 10) partitions the events into two clusters of equal 
size. The next element, "decisions based on incorrect weighting 
of sensory inputs" (number 15) , partitions each cluster created 
by element 10 into nearly equal subclusters (independent of 
element 10) . This partitioning is continued by the other crit- 
ical elements in sequence. An inference that can be made from 
the foregoing is that on the basis of this data set (past 
experience) , a future accident could be examined most efficiently 
by studying the critical elements in the sequence shown, and 
that the nine elements listed will most economically and quickly 
provide information on a new given non-training/non-midair acci- 
dent. It should be noted that the methodology described earlier 
for calculating likelihood for patterns of states of variables 
is also applicable to this data particularly when information 
regarding frequency of patterns that have occurred is available. 
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ACCIDENTS 


CRITICAL ELEMENTS 
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Critical Elements: 

Conditions : 

01. Experience 

02. Distraction 

03. Crew Coordination 

04. Neglect 

05. Air Traffic Control 

06. "Decisions" 

07. Work/Rest 

08. Machine 

09. Airport 

10. Weather 

Decisions based on: 

11. "Out of tolerance" conditions 

12. Erroneous sensory inputs 

13. Delayed decisions 

14. Make destination/press-on 

15. Incorrect weighting 

16. "Incorrect choice" 

17. Correct decisions 

18. Overloaded or rushed situation 

19. Desperation (self-preservation 

decisions) 


Figure 8. Clustered critical elements* on 56 non-training/ 
non-midair accidents. 

*1 = Present, 2 = Not listed. 
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OPTIMAL SEPARATION OF EVENTS BY CRITICAL ELEMENTS 
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PERCENT SEPARATION (S) 
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^ = Present 
FI = Not Listed 


Figure 10. Reclustered critical elements on 56 non-training/ 
non-midair accidents. 

♦Refer Figure 8 for listing of the 19 elements. 


The many different kinds of clustering methodology in 
themselves are barren. The crux of the matter is interpretation 
of results after clustering and, hopefully, that the clusters 
generated in the data set are well differentiated and will pro- 
vide corroborative evidence of being sensitive and stable. Each 
person is his own clusterer or classifier; there is no general 
theory of clustering or classifying anything. In parallel, 
there are many different kinds of clustering methodology and the 
open-minded investigator should entertain many alternative ways 
of handling his data set. In the discussion above, no so-called 
truth is ascribed to rearrangements of numbers automatically 
done by machines. A lower level of abstraction with more meaning 
is required - the statements of characteristics describing events 
are symbolized with numbers and are used in creating a compound 
definition of events. These are presented in combination with 
descriptions of other events thus allowing for new inferences 
and possible discovery. No fountainhead of truth in itself is 
implied, merely a form of descriptive statistics and the basis 
for inference. 



GENERAL DISCUSSION AND RECOMMENDATIONS 


A multidisciplinary human factors systems analysis has 
been conducted on all those NTSB records of U.S. air carrier 
jet accidents with human factors cause elements identified which 
were complete enough for exhaustive analysis. The time period 
covered was from 1958 through 1970. The human factor classifi- 
cation of the 74 accidents as assigned by the NTSB was accepted 
without question by the study investigators, but we have no 
quarrel with the classification of these accidents after ex- 
haustive study of the records. 

The analytic techniques applied here are new to accident 
analysis and provide a systematic evaluation of human factors 
data of such complexity as to be normally resistant to analysis 
by even sophisticated techniques (10). The critical decision 
and critical condition data are new and were not heretofore 
available or highlighted. The models for critical decisions 
and critical conditions are workable and offer promise for 
future accident evaluation. They have been most useful in 
identifying out of a morass of data those elements deserving 
further study and/or research. 

The NTSB computer data alone would not provide information 
on whether a given accident or set of accidents was human error- 
related, if that descriptor were not a part of the computer file 
for that accident. That is, the computer data have no selective 
qualities of identif iability from the viewpoint of critical 
elements we have outlined. This area is therefore deserving 
of further development. One could speculate that the relative 
barrenness of the data collection in the human elements of the 
computer filed information may have been due to a lack of 
satisfactory means of reducing the information to the computer 
language. In that case, the model presented here may help. 
Further, the model may add objectivity to the collection and 
interpretation of subjective information bits. The paucity of 
data could also be at least partly accounted for because of the 
lack of emphasis on such data collection or the absence of 
appropriate regard for the value of such data. Appropriate 
emphasis for all elements of data is imperative, with weightings 
due to individual bias on the part of the accident investigators 
precluded by the very nature of the investigative scheme. 

Human error versus non-human error partitioning may be 
felt to be a fundamental mistake in classification and it may 
be that a man-machine-environment or host- agent-environment 
scheme may be more fruitful, especially in longitudinal analyses. 
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However, "human error" as a classification, regardless of its 
shortcomings , has the overwhelming advantage of centering 
attention on the group of accidents (with at least one major 
common element) to which a significant amount of attention 
should be devoted, and to the group where a high likelihood 
of success exists. We have experienced no serious difficulty 
in applying the analytic techniques described earlier, which 
include the epidemiologic host-agent-environment scheme, as 
an overlay to the already categorized human error accident set. 

Inherent in the many difficulties faced by investigators 
and analysts who must attempt to make meaning out of collected 
bits of information is the changing quality of the contingencies 
affecting the accumulation of happenings and the gathering of 
data relating to these events. Observer variations, variations 
in the quality and quantity of data collected, variations in 
the techniques and efficiency of the coders of the data, are 
only a few of the problems. The nature of these problems is 
so complex that it would require a separate treatise to deal 
with this subject alone. In fact, many would point out that 
these very problems introduce such variability into the data 
that it is impossible to analyze and gather from the analysis 
any meaningful results or conclusions. 

Jacobs (10) , in a paper devoted to a discussion of some 
of the conceptual and methodological problems in accident 
research, deals particularly with problems encountered in 
formal statistical inference, when use is made of traditional 
techniques. He feels that the existence of the pitfalls and 
difficulties have contributed largely to the lack of research 
achievement in the field despite many years of effort. Pointing 
out that analysis of data on a large number of accidents in 
aggregate has rarely produced any practicable remedy which 
would 1 have prevented any substantial number of accidents, he 
attributes the problems of analysis to the fact that in any 
single accident there are a number of combined coincidental 
events and circumstances of a causal nature, in the absence 
of any one of which, there would not have been an accident, 
and that the production of the accident required the combination. 
To answer why analytical and correlational studies have seldom 
led to the development and application of useful countermeasures, 
he offers two explanations. First, he points out that this type 
of research presents very difficult methodological and conceptual 
problems, most of which are not commonly perceived, and, secondly, 
he states that the treatment of these methodological problems 
requires a magnitude of effort which goes far beyond that which 
is generally required for productive research in most other 
fields of inquiry. However, the data on accidents collected 
by the NTSB is virtually all of the information we have, and 
a system of interpreting such data, with all the inherent defects 
of the data admitted, is badly needed. The techniques used here 
provide, we feel, a method of dealing with an overwhelming mass 



of data, and a method of comparing apples and oranges without 
forcing the investigator into the hopeless corner of bit-by-bit 
analysis with which the human mind cannot cope. 

Another major problem which has plagued the entire field 
of accident investigation is that provided by the semantic 
restraints inherent in the definition of the term accident itself. 
Webster's Third New International Dictionary of the English 
Language (22) defines accident as follows (irrelevant comments 
are excluded for purposes of brevity) : 

accident n. (fr Latin root acoidens meaning nonessential 
quality or circumstance, accident, chance, from present 
participle of aocidere to happen) . la: an event or 
condition occurring by chance or arising from unknown 
or remote causes, b: lack of intention or necessity: 

CHANCE - often opposed to design, c: an unforeseen un- 
planned event or condition. 2a: a usually sudden event 
or change occurring without intent or volition through 
carelessness, unawareness, ignorance, or a combination 
of causes and producing an unfortunate result, b: an 
unexpected medical development especially of an un- 
fortunate or injurious nature occurring in apparently 
good health or during the course of a disease or a 
treatment, c: an unexpected happening causing loss or 
injury which is not due to any fault or misconduct on 
the part of the person injured but from the consequences 
of which he may be entitled to some legal relief. 

3: an adventitious characteristic that is either in- 
separable from the individual and the species or separable 
from the individual but not the species; broadly: any 
fortuitous or nonessential property, fact or circumstance. 

4: an irregularity of a surface (as of the moon). 

Syn: chance, quality. Syn for accidental : fortuitous, 

adventitious, contingent, casual, incidental. 

Note the synonyms for the words accident and accidental. 

The definition of the word causes semantic responses which 
are not fruitful - that is, the definition is inherently 
negative, both from the viewpoint of the victim and from the 
viewpoint o'f the person involved in dealing with the accident 
externally, such as the investigator trying to take a con- 
structive approach. Social acceptance of the phenomenon leads 
to apathy and a feeling of helplessness and futility in trying 
to deal with accidents. Since they are fortuitous, there is 
nothing we can really do about them anyway, so why try? 

Suchman ( 2 0) points out that the term "accident" implies un- 
expectedness, undesirability, with the implication that un- 
anticipated turns of events cannot be prevented and that there 
is little to do but clean up the damage. He further elaborates 
on the climate of helplessness and despair and attributes this 
climate to contributing heavily toward keeping this area of 
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human activity out of the realm of scientific investigation. 

He then goes on to point out that most accidents do not appear 
to be solely chance events, but compose a set of events which 
can be identified, classified and analyzed in terms of ante- 
cedent and associated events with an eventual goal being 
explanation of causation. 

Gibson, in discussing the contribution of experimental 
psychology to the formulation of the problem of safety ( 8 ) / 
refers to the term accident as a makeshift concept with a 
hodgepodge of legal, medical and statistical overtones. He 
states: "Two of its meanings are incompatible. Defined as 

a harmful encounter with the environment, a danger not averted, 
an accident is a psychological phenomenon, subject to prediction 
and control. But defined as an unpredictable event, it is by 
definition uncontrollable. The two meanings are hopelessly 
entangled in the common usage, there is no hope of defining 
it for research purposes. Hence, I suggest that the word be 
discarded in scientific discussion. The problem of accident 
prevention should be renamed - perhaps calling it the problem 
of safety. It is then, on the one hand, a matter of the 
ecology of dangers and the natural or artificial signs of 
danger, and, on the other hand, a matter of the psychology 
of their perceptions and reactions aroused by these signs. 

When thus reformulated, the problem appears in a quite different 
light." We wholeheartedly agree with this statement, and 
would add only that together with the discarding of the concept 
of accident as traditionally defined, should also go, once and 
for all, the concept of 'accident proneness 1 as a totally worth- 
less and unscientific concept. Suchman (20) points out that 
there are good reasons for disregarding this model on theoretical, 
methodological and mathematical grounds. He states that he does 
not believe that the complexity of accidents can be adequately 
explained by the simplistic model of a single personality type 
which seeks out accidents through some neurotic motivation. 

Arbous and McFarland ( 4,13) also find the approach quite in- 
adequate . 

We have not allowed ourselves to be drawn into the problem 
of defining an accident. The operational concept of this 
research project has been to deal with each "accident" as a 
successful event and we have found the concept useful in helping 
to establish a positive frame of mind, if nothing else. Viewing 
accidents as successful events, we have been able to review 
identifiable factors pertaining to the accident in terms of what 
we chose to call "accident enabling factors", or factors without 
which the accident could not occur. Not all factors (conditions 
and decisions) fit the definition of accident enabling factor, 
of course. Another pitfall we tried to avoid in the data 
collection phase was thinking of the events associated with an 
accident as being right or wrong, good or bad. They were simply 
viewed as events without judgment being attached. This is not 
to say that judgments are not possible or should not be made, 



but they were not the proper subject of the data gathering 
portion of the study. 

Accepting conditions out of tolerance (critical decision) 
and operating the aircraft in a condition off acceptable or 
optimal profile (critical condition) , while not identical in 
their definition, as can be seen above, are prominent in both 
the condition and decision analyses. Accepting conditions 
out of tolerance and operating off-profile are inherent in 
everyday airline operations. The training effect of operating 
outside of what is taught as "proper" airmanship may have a 
negative long-term value in flight safety. We feel that there 
is a need for intensive study into the relationships that exist 
between pilots, airlines and regulatory bodies, with special 
attention to defining and understanding how institutional 
decisions which involve all of these groups interact to affect 
flight safety. As an example of an institutional decision, 
we refer to an operational decision to continue a flight with 
one generator out of service. 

An over-all observation from the decision data leads us 
to recommend further research into the decision-making role 
of the pilot. The present cockpit environment has been shown 
to be less than ideal for a pilot to be a reliable decision 
maker. There are many proposals in the literature for either 
adding a pilot decision-maker (19,23) or restructuring the present 
pilot functions to include adaptive, computer-assisted decision 
making ( 7 ) . Resistance to change of the present cockpit work 
situation must be met by factual data which point to past 
deficiencies in pilot information processing. Task off-loading 
and actual training of pilots in decision-making techniques 
(particularly with regard to "critical elements") seem to us 
to be the most fruitful approaches. We are not prepared at 
this time to make recommendations pertaining to the introduction 
of specific new equipment into the cockpit to automate some 
pilot functions. The reason for this is two-fold; first, we 
have not satisfied our own requirements with regard to the depth 
of analysis needed in this area, but may have recommendations 
later. Secondly, others have pointed out that addition of new 
equipment may not necessarily simplify cockpit duties. Addi- 
tionally, the cockpit routines which are changed by introduction 
of new equipment may complicate the pilot’s job and require 
changes in learning patterns and instruction which would have 
to be analyzed for over-all impact on the man-machine-environment- 
contingency model. Obviously, this process should be carried 
out in relation to each specific piece of equipment, and is not 
the subject of this study. Improving our knowledge of the pilot 
as a decision maker could be enhanced by developing human factors 
criteria for over-all flight crew-airplane system performance; 
by improving feedback loops from on-the-job decision making to 
training requirements; by determining the amount and type of 
decision information a pilot needs for computer-assisted informa- 
tion processing and developing the criteria for the display 
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system; and attempting to determine control input task un- 
loading during critical flight regimes, with the captain 
becoming a decision initiator and inhibitor who provides 
occasional start and override commands. 

Various elements of the training environment of the 
pilot could well be enhanced by knowledge of the information 
uncovered here. For example, in the 56 non-training/non-midair 
accidents, decisions most often involved conditions out of 
tolerance followed by erroneous input information, incorrect 
weighting of inputs and requirements to make the destination 
or schedule. Training accidents most often involved decision 
delayed beyond the time required. Midair collisions most 
often involved decisions based on requirements to make the 
destination airport or meet the schedule. This information 
points to the recommendation that airline training and 
operational procedures must be updated to improve the re- 
liability of the pilot as an information processor. 

Some specific recommendations are made with regard to 
recognizing needs in accident investigation. First, there 
is a need for better information collection at the accident 
site. This information collection must be systematic and 
subject to verification. It also requires the insured 
cooperation on the part of all persons conducting the in- 
vestigation. This naturally implies that the Investigator- 
in-Charge must be attuned to the human factors potential for 
each accident and must insure that data collection is not 
impeded. Human factors investigation personnel are too often 
concerned with required hospital investigations, visits to 
mortuaries and other relatively routine matters while human 
factors information is being collected by operations and ATC 
groups. We do not ho'ld that operations and ATC investigators 
are not competent to collect human factors information, but 
we do believe that the human factors investigator must be 
constantly apprised of developments. Not only must the 
Investigator-in-Charge be highly sensitive to the needs in 
the human factors area and insure proper collection of data, 
but the Accident Inquiry Manager must also have the same 
orientation. Secondly, there is need for more information 
to reconstruct what transpired in the cockpit prior to the 
crash. The marked gaps in accident data are not only gaps in 
technical information, as seen by perusal of the matrix, but 
also significantly large gaps in human error material. The 
cockpit voice recorder was available in only seven of the 
74 accidents studied here. The cockpit voice recorder is 
the best single source of information on the accident and 
provides an objective means of judging the cockpit activities 
prior to the crash. We are aware that there are many reasons 
why the accident data are missing, but are also aware that a 
better collecting system would fill in many of the gaps in 
the future and would result in the collection of more pertinent 



human error material. Cockpit voice recorders are still being 
damaged, especially in those crashes where serious fires occur. 
Further research is needed to insure indestructible cockpit 
voice recorders. Thirdly, there is a need for a system to let 
all individuals directly involved in an accident give an 
unimpeded and straightforward story of what transpired. There 
is such an overriding consideration of legal liability and 
vested interests that pressures on all persons directly involved 
often result in a self-preservation decision which diminishes 
the reliability of information. As long as legal liability is 
determined from a single investigation, this problem will 
continue to present itself. We recommend that consideration 
be given to congressional legislation which would allow parallel 
investigations with regard to cause and legal liability. We 
submit that, in the long run, cause is more important than 
liability. Fourthly, there is a need for developing a system 
to allow quick determination of the likelihood that an accident 
has prominent human factors problems, so that such problems 
may be recognized and investigated in depth, with the least 
amount of time spent on other unproductive activities. 

The computer-stored information on the human error 
accidents contains objective, measurable characteristics such 
as runway length, ceiling, visibility, and pilot time. To 
this computer information must be added subjective, sensitive 
descriptions of the flight environment. When we were confronted 
with the size and bulk of accident information data, the 
technique devised to overcome this obstacle was for a trained 
researcher to read the accident report and describe those 
characteristics which became accident-enabling features. Most 
of the information derived with this technique were not available 
in computer- stored information. A good example of this was 
heavy rain at threshold. Such qualitative data may be recorded 
in the report, but for computer- stored information, no indication 
of concentration of rain was given. Neither was the position 
of the rainshower in relationship to the airport or the runway 
environment. Another characteristic observed with this- technique 
was off-profile flying that was either high or low, fast or 
slow, or offset. Again, this information was reported, but no 
capability existed in computer storage of this information. 

Few groups or individuals charged with accident analysis 
or investigation have been able to overcome the tendency to 
put greatest emphasis on the accidents associated with death 
and/or destruction. It is understandable that public attention 
is focused on the tragic accident, but we cannot agree that only 
those accidents or incidents deserve such attention. We were 
only able to use 74 of the over 200 accidents having human error 
associations because of the paucity of data in the non-tragic 
cases. The study of accidents involving human error should be 
justifiable on the grounds of the value of the information 
gained, and should not be tied to tragic outcome. Just as 
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important is the collection and study of data related to 
near misses, and we feel a wealth of information is available 
here. This would be especially so if anonymity were guaranteed 
and comprehensive systematic investigations were carried out 
after design of experimental techniques and hypotheses to be 
tested (17) . All parties to a near miss are available, unlike 
many accidents, and the information could prove invaluable. 

Although not specifically shown in the presentation of 
matrices selected for this paper, we have noted a tendency for 
aircraft accidents to occur early in duty periods. Also 
associated was the observation that a number of crews involved 
in accidents had returned to duty after 16 or more hours off 
duty and that there is a paucity of accidents noted at the 
home domicile of the crews. It has been suggested that there 
may be some relationship to the "warmup" phenomenon seen in 
industrial accident studies. Further elucidation of this 
subject is indicated, but in order to firmly establish whether 
the data are meaningful, it will be necessary to determine 
what leg in the duty period the crew was in at the time of 
the accident, the number of landings, etc., in order to 
determine, among other things, the relative risk of accident 
at the home domicile. 

Practical application of the concepts developed in this 
paper is desirable. This could take several forms: 

1. Testing of the conditions-decisions models for 
prediction capability, 

2. Use of the concepts to develop an expanded human 
factors accident investigation format, 

3. Use of the models for further data evaluation. 

This could require data expansion and, of course, inherent 
in the use of the technique is ongoing refinement and 
improvement as an inescapable function of use, 

4. Use of the models in simulator application. 



CONCLUSIONS 


A multidisciplinary team approach to pilot error-related 
aircraft accident investigation records has successfully re- 
claimed hidden human error information not currently shown in 
statistical studies of accidents. New analytic techniques have 
been developed and applied to the data in efforts to discover 
and identify multiple elements of commonality and shared char- 
acteristics within this group of accidents. 

Three techniques of analysis were employed: (1) cluster 

analysis and pattern recognition, (2) critical conditions, and 
(3) critical decisions. 

A cluster analysis technique has been modified and applied 
to the data. This method is an effective exploratory research 
tool that will lead to increased understanding and improved 
organization of known facts, the discovery of new meaning in 
large data sets, and the generation of explanatory hypotheses. 

The technique also provides a simple automated straight-forward 
means for visually presenting an otherwise unmanageably large set 
of data in its entirety in matrix form, for organizing the data, 
clustering the accidents and disclosing and summarizing the 
structure of the matrix with a minimum loss of information. 

Initial results of the critical element analysis portion 
of the study have demonstrated the importance of a subjective 
qualitative approach to raw accident data which has surfaced 
information heretofore unavailable in succinct detailed summary 
in records. Using this approach to extract and analyze those 
causal elements, attributes, events and conditions which signi- 
ficantly contributed or were related to the occurrence of the 
selected accidents, a model was developed. This model, consist- 
ing of critical conditions and critical decisions, was then 
applied to the training, midair, and non-training/non-midair 
groups of accidents. Other methods of analysis included the 
use of a "critical condition screen". 

Findings, conclusions and recommendations resulting from 
this study include: 

1. The cluster analysis technique presents a feasible 
method for establishing meaningful concepts and interrelation- 
ships out of a large mass of data otherwise impossible to 
mentally handle. 
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2. Critical element analysis supplements and adds 
perspective to accident data and provides vital insight into 
real-life operational human factors data in a format amenable 
to meaningful analysis. 

3. An aircraft accident, in our view, is a positive, 
eminently successful event that is inevitable when preceded by 

a series of one or more "accident-enabling factors" or conditions 
which have to be present and without which the accident cannot 
occur. This approach de-emphasizes pilot "fault" and focuses on 
the actions and decisions (both proper and improper) that are 
carried out as well as those factors that prompt these actions 
and decisions. 

4. Human factors data as currently collected are inadequate 
in defining the "why" of pilot error and the terminology and 
classification gives no insight into corrective measures. 

5. A number of areas have been identified that require 
more research. These include such items as crew interrelation- 
ships (coordination, interaction), decision making, approach 
monitoring, aircrew fatigue (work/rest schedules) , and subtle 
operational indices of crew stress and performance decrement. 
Current airline pilot training does not seem to adequately prepare 
the pilot for the situations encountered in the critical elements 
identified in this study. Furthermore, details of cockpit 
activity and events are not known in sufficient detail; cockpit 
voice recorder data do not provide needed information. 

6. There is a need for intensive study into the relation- 
ships that exist between pilot, airlines and regulatory bodies, 
with special attention to defining and understanding how insti- 
tutional decisions which involve all of these groups interact to 
affect flight safety. 

7. Airline training and operational procedures must be 
updated to improve the reliability of the pilot as an information 
processer. 

8. There is need for improvement in accident investigation, 
to include: 

a) More systematic collection of human factors data at the 
accident site. 

b) More information to reconstruct what transpired in the 
cockpit prior to the accident. 

c) Improved reliability of information by reassessing 
pertinent rules and principles of legal liability. 

d) Development of an on-site system to allow quick deter- 
mination of the likelihood that a given accident has prominent 
human factors aspects. This could lead to a more intensive 
investigation of human factors features on site, when the 
information is still fresh. 

9. More attention should be focused on the human error 
aspects of near-accidents and incidents. 



APPENDIX A 


CLUSTER ANALYSIS AND PATTERN RECOGNITION - 
BASIC METHODOLOGY 


Nature of the Data Set 

The data set consists of information collected on aircraft 
accidents over a period of 12 years. Real data sets of this 
type present unique problems of analysis. Development of statis- 
tical theory generally assumes the use of continuous variables on 
an interval scale. The data set we considered, however, consists 
of mixed variable types. For this reason, and the desirability 
of scale conversion of variables, brief mention will be made of 
variables and scales. 


Usually in mathematics, variables are classified as being 
binary (dichotomous), discrete, and continuous. Examples of 
these types of variables in this data set are: 

1. Binary variables: 


a. Runway lighting 

on 

Off 

b . Precipitation 

yes 

no 

c. Condition of light 

day 

night 

Discrete variables: 

a. Number of engines 

two three 

four 


3 . Continuous variables : 

a. Pilot age 

b. Temperature 

Variables, in addition, may be classified on the basis of their 
measurement scale as nominal, ordinal, interval, and ratio. 
Examples in this data set include: 


1. Nominal (x_ = x^ or x a ^ x^) : 
a. Precipitation 


2. Interval (if x a > x^, a is x a - x^ units or greater 
than b) : 
a. Temperature 


In turn, variables classified by range (binary, discrete, contin- 
uous) and by scale (nominal, ordinal, interval, and ratio) may 
be cross-classified. For example, Pilot 1 may be older than 
Pilot 2 or Age Pilot 1/Age Pilot 2 means Pilot 1 is so many times 
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older than Pilot 2, or Age Pilot 1 - Age Pilot 2 means Pilot 1 is 
so many years older than Pilot 2. By scale classification the 
variable Age is a ratio variable and by range classification the 
variable Age is a continuous variable. 

The pilot is continuously confronted with mixed variable 
types, including interval, ordinal, binary, and other types. 
Statistically, as stated previously, an assumption is made 
usually that variables are continuous and on an interval scale 
of measurement. Many of the variables in this data set fulfill 
this requirement; other variables as illustrated do not. In an 
attempt initially not to weight the variables in the data set 
as to their relative importance as possible direct or indirect 
causative factors in pilot error-related events, and not to 
exclude variables arbitrarily from the data set as being unim- 
portant, variables were scale-converted to what may be called 
polychotomous variables . For example, airport elevation at which 
accidents have occurred, or the total range of airport elevation 
in the U.S., might be sea level (0*) to 7200". This continuous 
variable may be divided into intervals of sea level to 100', 

101* to 2000', and over 2000'. Each interval of the variable 
may be given a "state" or value of logic designation. That is, 
0-100* designated by 1, 101-2000* designated 2, and greater than 
2000* by 3. One or more variables within the same data set may 
be scaled into different numbers of states which are given a 
"value or state of logic." This permits the construction of 
"truth tables", which consider every possible combination of 
logic states. For example, if all variables were scaled to 
binary (dichotomous) values (two states or values of logic) and 
there were 1, 2, 3, 4, etc., of these variables, the total 
possible number of binary sequences of yes and no (1 and 0) 
would be respectively 2 1=2, 2^=4 , 23=8, 2^=16, etc. (See Table 
12.) An important observation is that the truth tables of the 
higher valued states always contain the sequences of the lower 
valued states, i.e. , truth table C is contained in truth table E 
(Table 12). 

This methodology then permits the analysis of mixed vari- 
able types in the same matrix, closely approximating the real 
world data set. After the variables have been scaled, the data 
are analyzed by clustering, by truth table analysis to determine 
the frequency of patterns of variables. This enables one to 
establish the likelihood of occurrence of patterns as states of 
variables that contribute to pilot error-related events. Lastly, 
the method permits the selection of decision variables to be 
applied in studying the next event to determine if it fits pre- 
vious variable patterns. The methods used are forms of matrix 
or concept learning which are correctable as more information 
becomes available with time. Statistical stability of clusters 
detects states of variables which may be correctable states for 
improving flight safety. 

By use of automated data processing procedures it is 
possible to handle large data sets. For purposes of illustration 



TABLE 12 


Examples of 2-state (A - D) and 3-state (E) conversion of 
variables and the possible sequences. These are truth tables 
and are Boolean expansions of Aristotelean truth tables. 


Truth Table: A B 

2 1 2 £ 

1 11 

0 10 

01 
00 


c 

D 

E 

23 

2 4 

_3i 

111 

1111 

222 

110 

1110 

221 

101 

1101 

220 

100 

1100 

212 

Oil 

1011 

211 

010 

1010 

210 

001 

1001 

202 

000 

1000 

201 


0111 

200 


0110 

122 


0101 

121 


0100 

120 


0011 

112 


0010 

111 4 — 


0001 

110 >4 — 


0000 

102 

101 ■* — 
100 4 — 
022 
021 
020 
012 

Oil 4 — 
010 4 — 
002 

001 4 

000 4 — 


*Arrows indicate where truth table C sequences are contained 
within truth table E. 
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to show how the methodology described can be used to draw con- 
clusions concerning contingency relationships in accidents, a 
small and generalized example will be given using a total of 27 
accidents and four variables scaled to 3-valued logic. The 
nature of the data set is such that data are collected as they 
become available in time, with the occurrence of each event. 

Data on selected variables are systematically collected as well 
as data for any other variable which appears applicable. The 
data are presented in the format of a matrix of events-versus- 
variables with the state of the variable indicated at the proper 
cell in the matrix. 

Initially, data may be recorded sequentially for 13 events 
over four variables scaled into 3-valued logic (2, 1, 0), with 
"2" assigned the highest value of logic (see Table 13) . For 
each variable, over all events, the numbers (N) of each state of 
logic are enumerated (N 2/ N^, Nq) (Matrix A, Table 13). The 
columns (variables) are then rearranged in descending order of 
the highest level of logic (leftmost column has greatest total 
of "2's") (Matrix B, Table 13). Each row pattern in Matrix B 
is traced on the truth table in Table 14 (solid black square = 2, 
open square with dot = 1, open square = 0) and the rows of 
Matrix B are rearranged vertically in their order of occurrence 
in Table 14. Three large clusters of five, three and five events 
(outlined with lines) are formed, each having subclusters 
(Matrix C, Table 13). 

The reason for applying this clustering methodology is that 
all clusters and subclusters are logically disjoint. This means 
that when the next event occurs the pattern of states of the 
variables may be identified uniquely as belonging to a particular 
cluster and the same as some event in the cluster. Classification 
(synonomous here with the formed clusters) and identification are 
considered inverse mental activities. Clustering or classifying 
and the clusters and classes so formed are considered a matter 
of induction because we do not have every possible event and we 
infer that the classes or clusters are stable and significant. 
Inversely, identification is considered a matter of deduction . 

If the next event is not identifiable, however, the values Tor 
the variables are added to the matrix and the resulting updated 
matrix reclustered. This process is shown in Table 13, Matrices 
D, E, and F. Clusters in Matrices C and F may be summarized. 

This is sometimes preferable, especially when the number of 
events becomes very large. These clusters are formed by "relat- 
edness" of states of variables and provide insight for discover- 
ing possible new heretofore unsuspected relationships within 
these events (accidents) . An advantage of using clustering is 
that known data are all considered within one matrix. These 
clusters demonstrate a structure in the data set and the rela- 
tionships of patterns of variables to events, with a minimum 
loss of information. This would not be possible if the data were 
abstracted further. In addition, as data are accumulated, it is 
possible to determine the sensitivity and statistical stability 
of the clusters. This forms a basis for the study of relation- 
ships of variables and factors within the events. These are 
learning matrices, as described by Rypka (18). 



In Matrix B the variables (columns) are rearranged in descending order of the highest 
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An idealized illustration of multi-valued logic is shown. In this example, four characters 
having 3-valued logic (3 4 =81 patterns) are listed in the truth table at the left. These could 
represent 81 unique possible patterns of events. The event-versus-event matrix lists the com- 
parison of every pair of patterns. When values for a given variable' differ, the two events 
are separable or disjoint. In this ideal case, all pairs of patterns (events) are separable. 
The progressively smaller squares within the matrix represent the numbers of pairs separated 
by each variable conditionally independent of the other variables. The idealized curve for 
the percent separation of pairs, calculated using the general equation, is shown in the Table. 
Real data, clustered by the truth table method shown in Table 13, are made to approximate the 
ideal curve as closely as possible. 

*Rypka, E. W. Pattern Recognition as a Method of Studying Host-Parasite Interaction. 

1972 International Conference on Cybernetics and Society, Washington, D.C. 
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APPENDIX B 


CLUSTER ANALYSIS AND PATTERN RECOGNITION - 
DETERMINATIVE AND IDENTIFICATION METHODOLOGY 


The theoretical aspects of separation and partitioning of 
events are shown in Table 14. The truth table shown at the left 
consists of four 3-state variables (3-81 possible patterns of 
results) . Each row is compared with every other row, and the 
variable (s) that separate (s) a given pair of rows (events) 
(actually the event's "partitioning variables") is recorded in 
the " event-versus -event " matrix. In this case there are e(e-l)/2 
(e=event) or 81(80)/2 = 3240 pairs of events to separate. We 
wish to analyze the next event in terms of optimal selection of 
variables so that an identification can be made in the most 
accurate, systematic, rapid and economical manner. 

The general identification equation is: 

S = [1 - v _k ]/[l - v" c ] 

S = separation value for the variable 
v = state or value of logic (3 in this case) 
k = kth character or variable 
c = number of characters or variables 

An example of the equation is used in Figure 11. It is necessary 
next to select variables (for example, from Matrix C and Matrix F, 
Table 13) so that the pairs of events are separated at a rate that 
most closely approximates the general case. To do this, variables 
are selected in sequence so that the first variable separates the 
most pairs. The second variable separates the most pairs condi- 
tionally independent of the first selected variable, etc. Ideally 
every combination of variable, C£, should be tested but hardware 
limitations soon become evident Because of the large number of 
combinations to be studied. The empirically determined variable 
selection and rate of separation of pairs of events for Matrix C 
of Table 13 are shown in Table 15. 

When sufficient data become available, it is then possible 
to assume that each variable is statistically independent of the 
others and, with optimal selection of variables as just discussed, 
to calculate the likelihood for each pattern of variables. This 
provides a basis for predicting the likelihood of human error- 
related events for patterns of various states of variables. It 
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General Identification Equation 
S = [l-v -k ]/[l-v-c] 

v-3 , c=4 

Matrix C *g 

0.718 
0.910 
0.952 
0.987 


*Table 14 illustrates an ideal case. 


Theoretical # 

0.675 

0.900 

0.975 

1.000 


Figure 11. The theoretical rate of separation or partitioning 
of events and separation based upon data in Matrix C of Table 13. 
Variables selected to separate events that most closely fit the 
theoretical curve means identification of future events is being 
done at the fastest rate. 
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TABLE 15 



Variable selection in an event-versus-event matrix. Data are 
from Matrix C, Table 13. Each event in the matrix is compared 
with every other event. Variable states that separate events 
are entered in the squares in this table. Variable selection 
for identification is done so that the first best variable is 
the one separating the most pairs of events. The second vari- 
able is the one separating the most pairs of events conditionally 
independent of the first variable, etc. This optimal partition- 
ing is shown in Figure 11. 
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also allows for avoiding and for correcting against combinations 
of variable states , based upon previous experience, that were 
present in accidents and have a high likelihood of occurrence. 

To illustrate this method, using variables from our data set, 
four variables were selected from the matrix of the 56 non- 
training/non-midair accidents found in Figure 4. (The method is 
also illustrated elsewhere in this report on critical element 
data. ) 


Table 16 summarizes the steps taken in computing separation 
or partitioning information on the 49 variables. The number of 
logic states for each variable in Table 16, over the 56 events, 
is listed. Five-valued logic (1 through 5, in descending value 
of logic) was used. Logic states "8" ("not applicable") and "9" 
("no data") were omitted from the calculation of the S value for 
each variable. The value S represents the number of pairs of 
events a given variable will separate. The lower two lines in 
the table list the variables rearranged in descending order of 
their S values. Arbitrarily, the first four variables (numbers 03, 
16, 17 and 18), in 3-valued logic, are selected to illustrate the 
capability of predicting "likely and unlikely" combinations of 
variable states (patterns) , based on accumulated data. The 
variables selected are listed below: 



Variable 

Logic Value 

03 

Phase of operation 

i Landing 

1 



Approach 

2 



Departure 

3* 

16 

Total flight time 

Under 3500 

1 



3500 - 10,000 

2 



Over 10,000 

3 

17 

Hours in type 

0 - 500 

1 


(Captain) 

501 - 2000 

2 



Over 2000 

3 

18 

Hours in type 

0 - 500 

1 


(First Officer) 

501 - 2000 

2 



Over 2000 

3 

*In 

this example, "3" 

includes inflight, departure, 

takeoff to first power reduction and static/ramp. 

From 

the summary of the 

data set in Table 16, the 

frequency 

occurrence of the states 

for the variable are: 



Logic State 

Variable 




03 16 17 

18 


1 

0.429* 0.250 0.327 

0.473 


2 

0.250 0.339 0.454 

0.400 


3 

0.321 0.411 0.218 

0.127 


*N 1 /N 1 + N 2 + N 3 
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COMPUTATION OF S-VALUES 
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the number of pairs of accidents a given variable partitions 
riables rearranged in descending order of S-value 
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634 614 611 39? 56? 547 53? 47* 468 467 430 4*5 421 395 339 325 *97 *93 242 


Then, the likelihood of the following patterns of variable states 
would be: 


Variable Likelihood Normalized* 

03 16 17 18 

Logic State Pattern 3133 

Frequency (0. 321) x (0. 250) x (0. 218) x (0. 127) = 0.0022218 0.0554274 


Logic State Pattern 1321 

Frequency (0. 429) x (0. 411) x (0 . 454) x (0. 473) - 0.0378630 0.9445725 

Sum 0.0400848 

*By dividing the likelihood by the sum of likelihoods. 


From the preceding normalized likelihoods the inference would be 
made that variable states pattern 1, 3, 2, 1 for the four vari- 
ables 03, 16, 17 and 18 has a high likelihood of occurrence and 
that the information in this pattern should be further investi- 
gated. 


In summary, by clustering, optimally selecting variables, 
and calculating likelihood, it is possible, using real data sets 
to gain insight into combination of variable states, to determine 
likely and unlikely combinations thus providing some degree of 
predictability to pilot error-related aircraft accidents. 
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